Method and system for user interface for interactive devices using a mobile device

ABSTRACT

A software application and system that enables point-and-click interaction with a TV screen. The application determines geocode positioning information for a handheld device, and uses that data to create a virtual pointer for a television display or interactive device. Some embodiments utilize motion sensing and touchscreen input for gesture recognition interacting with video content or interactive device. Motion sensing can be coupled with positioning or localization techniques the user to calibrate the location of the interactive devices and the user location to establish and maintain virtual pointer connection relationships. The system may utilize wireless network infrastructure and cloud-based calculation and storage of position and orientation values to enable the handheld device in the TV viewing area to replace or surpass the functionality of the traditional TV remote control, and also interface directly with visual feedback on the TV screen.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. provisional application Ser.No. 61/555,989 filed Nov. 4, 2011, entitled “Method and System for anIntuitive User Interface to the Television Screen Using a Smartphone orTablet PC Device for Recognizing Gestures and Providing Visual FeedbackOn the TV Display,” which is incorporated herein by reference in itsentirety.

TECHNOLOGY FIELD

The embodiments of the present invention relate to the field of wirelesspointer interfaces. More specifically the present invention relates to asystem and method for establishing virtual pointer connections between aTV screen and a GPS-enabled handheld device for TV remote control orallowing a handheld device to act as a remote control for otherinteractive devices.

BACKGROUND OF THE INVENTION

The modern video viewing experience within consumer households has beenchanging dramatically in recent years, with more interactivity beingintroduced for content navigation, social media, T-Commerce, and moredisplay output options becoming available in the home. In manyhouseholds multiple TV displays are now complemented by laptops, PCmonitors, and smaller displays like tablet computers and smartphones forvideo viewing throughout the home, and on the go. Smartphones arestarting to be used to navigate video content, schedule and manage DVRrecordings, and as remote control devices for connected TVs. While somesystems may allow a smartphone or other network device to manage set topboxes via web-protocols, the interfaces are generally web-driven.Televisions and other traditional video displays currently lack thepoint-and-click user interface that personal computers have madeavailable to their users for decades. Applicant is unaware of anycommercially viable approach to turning generic TV monitors intointeractive displays without connecting them to computers (e.g.,displaying the output of a PC to a TV monitor), or embedding them withcomputer and sensor components (e.g., providing a console device with acamera or light bar and dedicated controllers for controlling theconsole, such as the Wii game console offered by Nintendo, which usededicated controllers with cameras to sense the direction andorientation of lights provides by an IR light bar controlled by theconsole and placed at a fixed location relative to the TV screen.)

Several approaches have been proposed for making TV remote controldevices behave more like a computer mouse, but none have proven to beconvenient for the consumer or cost effective for CPE manufactures orPayTV service providers. For the video gaming community the industry hasmade great strides with the introduction and adoption of devices likeNintendo's Wii Wand and Microsoft's Kinect interactive camera system.These motion sensing methods for remote control of gaming systems arestarting to bridge the gap between TV and PC human-machine interfaces,but they require the addition of expensive hardware into the TV/Gamingarea in the home. Apple's success with the iPhone and iPad introducedthe smartphone and tablet as new form factors for remote controldevices. TV remote control mobile apps are available for smartphones andtablets that provide a traditional remote control interface on thetouchscreen of the device, which may interact with the homeentertainment components via an IR emitter on the phone or via aBluetooth or network link to specialized set-top devices that provide anetwork control interface. These set-top devices may be provided bycable operators and may allow network devices to schedule or managerecordings, and in some cases allow real-time interaction with the cablebox by receiving remote-control like commands over the network. Theseinterfaces have been limited, however, to the traditional remoteinterface of up-down-right-left, numeric entry, and selection ofpre-defined remote buttons. In some instances, selection of individualshows or recordings can be made using the touch screen of the smartphoneor tablet, rather than by interacting with an interface on the TV.

Today's smartphones and tablets come with onboard sensors that can beuseful for locating rough location and orientation of the device.However, these sensors generally lack the precision needed to utilizethis information for providing a user interface. For example, GPSsensors are useful for providing a rough approximation of the locationof the device, to allow the phone to be located within several meters.This may be sufficient to locate the phone near an address whilenavigating in a car or looking for nearby restaurants. However, thesesensors have thus far been unable to replace the sensor used on othersystems that provide home entertainment interfaces (such as cameras withIR range finders and IR beacons or ultrasonic devices that are hardwiredinto the TV or console devices.) Accordingly, there remains a disconnectbetween the capabilities of mobile devices, such as smartphones andtablets, and providing a truly interactive interface with TV displays orother interactive devices.

SUMMARY OF THE INVENTION

According to embodiments of the invention, mobile handheld devices, suchas smartphones and tablet computers, can sense their position, location,direction, and motion in space, and can be programmed to recognizegestures. These devices can function as a virtual pointer andcontrollers with respect to their environment and devices such astelevision displays or home automation devices. These mobile devices canbe programmed via their operating systems and apps to communicate withone or more servers that can communicate with devices, such as set topdevices or home automation devices, that control the TV or any otherdevices that the user of the mobile device wishes to control or interactwith. By providing methods to improve the precision used in calculatingthe location of the mobile device within its environment, an improvedmodel of where the device is pointing in space can be created. Byapproximating the location and direction of the device, a system ormethod can determine where on a screen the user is pointing his mobiledevice or what interactive device a user is pointing a mobile device at.In some embodiments, a precise or approximate mathematicalrepresentation of the TV screen location (or location of any otherobject to be interactively controlled) can be calibrated and used tocorrelate an object or section of a display screen with the direction auser points a mobile device. In some embodiments, motions of the mobiledevice may be interpreted as gestures, allowing a mobile device to beused as a motion and gesture sensing remote control or virtual pointer.

Embodiments of the present invention address and overcome one or more ofthe above shortcomings and drawbacks by providing devices and systemsfor determining the position and orientation of handheld wireless/mobiledevice, such as a cell phone or tablet, and providing an app on thedevice to allow the mobile device to interact with a video display orinteractive device, such as home-automation devices.

According to one embodiment of the invention, a system for interactingwith a video display includes non-transitory computer-readableinstructions for configuring a processor of a handheld wireless deviceto determining position and orientation information about the handhelddevice and sending the position and orientation information across atleast one wireless network, and at least one server configured toreceive the position and motion information and to determine theorientation of the handheld wireless device relative to a the videodisplay. The position information can be derived from a distancemeasurement between the handheld wireless device and at least onewireless networking device, such as a Wi-Fi hotspot or a cell tower. Theserver can be configured to facilitate user interaction with a userinterface on the video display in response to the position andorientation information.

According to one aspect of some embodiments, a set-top box, incommunication with the at least one server, displays the user interfaceon the video display and updates the user interface in response tochanges in the position and orientation information. According toanother aspect of some embodiments, a server includes software operatingon the set-top box. According to another aspect of some embodiments,non-transitory computer-readable instructions determine motioninformation relating to the position and orientation information torecognize gestures made by a user. According to yet another aspect ofsome embodiments, non-transitory computer-readable instructions allow aprocessor to recognize at least a subset of the following gestures:Select, Move Up, Move Down, Move Left, Move Right, Back, Forward, ScrollUp, Scroll Down, Scroll Left, Scroll Right, Hover, Stop, Play, Pause,Fast Forward, Rewind, Record, Save, Last, Channel Up, Channel Down,Volume Up, Volume Down, Mute, Hide, Exit, Off, and Help.

According to one aspect of some embodiments, position information isderived from GPS signals received by the handheld device. According toanother aspect of some embodiments, the user interface presents apointer on the video screen that moves in response to movement of thehandheld device. According to yet another aspect of some embodiments,non-transitory computer-readable instructions display, on a display ofthe handheld device, at least one menu that supplements or duplicatesinformation displayed on the video display. According to still anotheraspect of some embodiments, at least one other wireless networkingdevice comprises at least one Wi-Fi access point.

According to one aspect of some embodiments, distance measurement isdetermined by observing a form of radio frequency time of flight orcommunication time of at least one signal sent to and received from theat least one wireless networking device. According to yet another aspectof some embodiments, the server is further configured to facilitateinteraction with a plurality of video displays, and to determine whichof the video displays the user interacts with based on the position andorientation information. In some embodiments, the handheld wirelessdevices include one of a tablet computer and a cell-phone.

According to one embodiment of the invention, a system for facilitatinginteracting with an interactive device in a physical space includes afirst set of non-transitory computer-readable instructions forconfiguring a handheld wireless device to determining position andorientation information about the handheld wireless device andtransmitting the position and orientation information and at least oneuser input across a wireless network, a second set of non-transitorycomputer-readable instructions for configuring at least one processor toreceive the position and motion information and to approximate theorientation of the handheld wireless device relative to a physicalspace, and a third set of non-transitory computer-readable instructionsfor configuring at least one processor to facilitate an interactionbetween a user and at least one interactive device in response to theposition and orientation information and at least one user input. Theposition information can be derived from a distance measurement betweenthe handheld wireless device and at least one wireless networkingdevice, such as a cell tower or Wi-Fi device located within the physicalspace.

According to one aspect of some embodiments, the interactive deviceincludes a video display. According to another aspect of someembodiments, the third set of non-transitory computer-readableinstructions are configured to be executed by a home automation serverand at least one interactive device comprises at least one of a lightswitch, an alarm device, and an HVAC device controlled by the homeautomation server. According to another aspect of some embodiments, thesecond set of non-transitory computer-readable instructions areconfigured to be executed by one or more servers in a cloud computingspace configured to communicate with the handheld wireless device acrossthe Internet. According to yet another aspect of some embodiments, thefirst set of non-transitory computer-readable instructions comprisesinstructions for displaying, on a display of the handheld wirelessdevice, selectable actions that can be taken to interact with the atleast one interactive device to solicit the at least one user input.

According to one aspect of some embodiments, selectable actionsdisplayed on the handheld wireless device change depending on which of aplurality of types of interactive devices that the points. According toanother aspect of some embodiments, the first set of non-transitorycomputer-readable instructions comprise instructions to recognizegestures made by the user to determine at least one user input.According to yet another aspect of some embodiments, the wirelessnetworking devices include at least one Wi-Fi access point. According tostill another aspect of some embodiments, the distance measurement isdetermined by observing a form of radio frequency time of flight orcommunication time of least one signal sent to and received from the atleast one Wi-Fi access point. According to one aspect of someembodiments, position information is further derived from GPS signalsreceived by the handheld device.

According to one embodiment of the invention, a method for facilitatinginteraction between a mobile device and an interactive device includessteps of observing, utilizing an antenna and one or more sensors on ahandheld wireless device, one or more distances relative to a pluralityof network devices in a space and an orientation of the handheldwireless device and receiving, across at least one network, locationinformation from the handheld wireless device that reflects the positionand orientation of the handheld wireless device within a physical space.The method may also include calculating a position and orientation ofthe handheld wireless device relative to at least one interactive devicefrom the location information to determine that a user is pointing thehandheld wireless device relative to the interactive device, determiningat least one command input from a user of the handheld wireless device,and carrying out the command on the at least one interactive device inresponse to the step of determining and the position and orientationinformation.

According to one aspect of some embodiments, the interactive deviceincludes a video display. According to one aspect of some embodiments,the interactive device includes at least one of a light switch, an alarmdevice, and an HVAC device controlled by a home automation server.According to another aspect of some embodiments, the step of observingincludes observing a GPS position of the handheld wireless device.According to another aspect of some embodiments, the position of atleast one interactive device can be calibrated by providingcomputer-readable instructions to allow the handheld wireless device torecord a position of at least one portion of the at least oneinteractive device when the handheld wireless device is placedapproximately at the position of the interactive device in the physicalspace.

According to one aspect of some embodiments, motion gesture made by auser of the handheld wireless device are detected to determine at leastone command input. According to another aspect of some embodiments, thehandheld wireless device detects a touch screen input made by a user ofthe handheld wireless device determine at least one command input.According to still another aspect of some embodiments, a cursor can bedisplayed on a video display that indicates approximately where a useris pointing the handheld wireless device.

Additional features and advantages of the invention will be madeapparent from the following detailed description of illustrativeembodiments that proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other aspects of the present invention are bestunderstood from the following detailed description when read inconnection with the accompanying drawings. For the purpose ofillustrating the invention, there is shown in the drawings embodimentsthat are presently preferred, it being understood, however, that theinvention is not limited to the specific instrumentalities disclosed.Included in the drawings are the following Figures:

FIG. 1 is a component diagram of the system hardware and devices used insome embodiments of the present invention;

FIG. 2 is a block diagram of the system software component modules usedin some embodiments of the present invention;

FIG. 3 is a logic flow diagram of the display setup and calibration usedin some embodiments of the present invention;

FIG. 4 is a calibration sequence diagram used in some embodiments of thepresent invention;

FIG. 5 is a high-level user interface and system components diagram foruse with some embodiments in the present invention;

FIG. 6 is a messaging communication sequence diagram illustratingelements of some embodiments of the present invention;

FIG. 7 is an infrastructure diagram representing some embodiments in thepresent invention;

FIG. 8 is an infrastructure diagram representing positioning features ofsome embodiments in the present invention;

FIG. 9 is a component diagram of the system hardware and devices used insome embodiments of the present invention;

FIG. 10 is a logic flow diagram of the display setup and calibrationused in some embodiments of the present invention;

FIG. 11 is a component diagram of the system hardware and devices usedin some embodiments of the present invention;

FIG. 12 is user interface diagram illustrating some interfaces for usewith some embodiments of the present invention;

FIG. 13 is user interface diagram illustrating some interfaces for usewith some embodiments of the present invention;

FIG. 14 is user interface diagram illustrating some interfaces for usewith some embodiments of the present invention; and

FIG. 15 is a logic flow diagram of a positioning algorithm for use withsome embodiments of the present invention.

DETAILED DESCRIPTION

Some Embodiments of the present invention address several of theshortcomings of the prior art by providing software methods and systemsthat allow a smart phone or a tablet device (i.e., a mobile device,which may also be characterized as a handheld wireless device) to act asa virtual pointer and remote control for interacting with a videodisplay or interactive objects in a physical environment, such as insideor outside a house or apartment. In some embodiments, software on themobile device utilizes existing sensors on the device to provide arelatively precise approximation of the location and orientation of themobile device within a physical environment. By approximating thelocation and direction of a mobile device, an approximate model of wherethe device is pointing in the environment can be created. Using themodel of the direction in which the mobile device points, software canfurther determine what the device points at. For example, by calibratingthe location and orientation of a TV display, when a mobile devicepoints to a certain location on the TV display, the location andorientation of the mobile device can be used to determine the portion ofthe TV display to which the user is pointing.

Furthermore, in some embodiments, relative motion sensors (such asaccelerometers) provided on typical mobile devices can be used to track,in real time, how a user moves the mobile device relative to the TVdisplay. This can be used to allow the mobile device to act as a virtualpointer or to recognize motion gestures. When a user points to a certainsection of the TV screen, the relative motion of the device can betracked, allowing a visual indicator of a cursor approximating where theuser is pointing to be displayed on the screen that tracks the motion ofthe mobile device as the user points to other portions of the screen. Inthis way, in some embodiments, the mobile device can act like a virtualhandheld mouse capable of interacting with a set top box that controlsthe output of a TV. Similarly, a quick swipe of the mobile device whenpointing at the TV can be identified using internal motion sensors andinterpreted as a gesture command to skip forward when viewing arecording on the TV or to change to the next item in an onscreen menu.

Additionally, in some embodiments, a screen on the mobile device can beused to provide additional information to a user or to act as anadditional input. For example, the screen on the mobile device candisplay a copy of the menu on the TV screen to allow a user to selectitems using the touch screen on the mobile device as an alternativeinput for selecting a menu item on the TV screen by moving a pointer tothat item. In some embodiments, the mobile device screen can displaybuttons that can be used when interacting with the screen, such as “OK”or playback buttons. In some embodiments, detailed information about ashow or recording, such as filmography, actors, ratings, reviews, etc.can be displayed on the mobile device while the TV displays a scheduleor plays back a recording. In some embodiments, the informationdisplayed can be updated as the user moves through menus or as he pointsthe mobile device at different portions of the screen. For example, asthe user points at different items in a schedule, the screen on themobile device can update to display information about each show orrecording to which the device momentarily points.

In some embodiments, a mobile device can interact with an interactivedevice other than a TV display. For example, a home automation servercan be used to make lights interactive. When a user points a mobiledevice at a lamp, the location and orientation of the mobile device canbe extrapolated to determine that mobile device is pointing at a knownlocation of a lamp. A display on the mobile device can be updated todisplay on/off buttons to allow the user to turn on or off the lights.Similarly, the mobile device can be used to arm or disarm an alarmsystem. A keypad can be displayed, or a simple on/off button can bedisplayed when a user points at an IP-enabled alarm panel.

Positioning

In one embodiment the Smartphone's location is synchronized with theTelevision Display location using a global navigation satellite system(GNSS) and differential Global Positioning System (DGPS) coordinates forprecision GPS Geocode readings. Civilian applications of GNSSs (e.g., USGPS) use CDMA coding in primarily the High frequency, VHF and UHF bandswhere the signals travel by line of sight, and in the US system NAVSTARsatellites transmit signals on one of two L-band frequencies (designatedas L1 [civilian and military use] and L2 [military use]). The L1 signalis modulated with a coarse-acquisition code (C/A-code), and a precisioncode (P-code), and the L2 signal is modulated with a P-code only. Todate P-code GPS applications have been limited to military use and canlog geocode (Lat/Long coordinates) to within centimeters of accuracyprecision for Satellites, ground based stationary or mobile receivers,and airborne receivers. Civilian GPS is only accurate to within metersfor real-time readings, unless augmented with a Differential-GPS (DGPS)technique. According to Garmin, their newer GPS receivers improveaccuracy to less than three meters on average from an earlier fifteenmeter accuracy benchmark. DGPS can use one or more terrestrialbase-stations with known locations to triangulate with three or more MEOsatellites. The U.S. Coast Guard's DGPS correction service uses anetwork of towers to transmit a corrected signal, making it the mostcommonly used DGPS service, but it is geared toward maritimeapplications without the need for centimeter-level precision.

Today's smartphones have a differential beacon receiver and beaconantenna already on-board in addition to their GPS receiver. Mobiledevices can derive an approximate location using an onboard GPS receiverto determine a position within a few feet using satellite signals.Furthermore, since smartphones and other mobile devices communicate withcellular towers and with other base-station type devices such as Wi-Fiwireless access points (WAPs) with known terrestrial stationarylocations, mobile devices can have access to reference points that canbe used for augmenting high-resolution precision DGPS geocodes withfurther accuracy. In some embodiments, this accuracy can be withincentimeters. In some embodiments, improved accuracy, includingsub-centimeter accuracies, may be achieved by utilizing longerobservation times. These observation times may utilize higherpost-processing requirements and may extend for hours. In someembodiments, post processing of location information from GPS and DGPSsignals can be performed outside the mobile device by sending theinformation gathered to an external server, such as a cloud-based serveraccessible via the Internet.

In North America alone there are over 200,000 cellular tower locations,and public WAP locations are growing to an order of magnitude moresites. In addition, the average home has several active Wi-Fi deviceswith a generally fixed position. For example, a wireless router in ahome typically does not move often once installed. Set-top boxes andvideo game consoles may use Wi-Fi, and sit in the same location near aTV once installed. In contrast, leading subscription DGPS services useroughly 200 tower locations globally, and as consumer applications ofDGPS for precision geocodes become common, cellular tower sites, and WAPlocations can be used for DGPS globally. Cellular tower operators,industry analysts, and service providers maintain databases of highlyaccurate geocodes of cell tower locations for wireless networkmaintenance and engineering. Nielsen Mobile, for example, publishes aCell Tower Locator Coverage Measurement report and database as asubscription service. Those skilled in the art can access cell towergeocodes and, in some embodiments, use those geocodes for DGPScorrection. In some embodiments, this data can be provided by telecomcarriers for the convenience of customers, possibly as a value addedservice. One skilled in the art may also get the geocodes used forcalculating location with the specific decimal-degree ordegree-decimal-minute format including pre-defined number of decimalintegers such that near centimeter precision resolutions can beachieved. This level of high resolution precision geocoding usingcivilian GNSS methods for real-time position measurement can enablemobile devices with virtual pointer functionality without the additionof new hardware in the TV viewing area, i.e., the consumer's livingroom.

In some embodiments, a technique, which may be referred to as secondaryor supplemental DGPS (SDGPS) can be used to refine the geocodecoordinates of a mobile device and any interactive devices or TVdisplays. In some embodiments a substantially real-time sequentialmultilateration (RTSM) technique, such as that shown in FIG. 15 can beused to provide SDGPS coordinates. In high-rise locations or in rural ormountainous locations the DNSS signal could be obscured, or DGPS beaconcoverage might not be dense enough to provide accurate DGPS readings. Asnoted in the previous embodiment the GNSS's RF signals travel by line ofsight and are subject to reflections or blockage and terrestrial GNSSreceivers require signals from four or more satellites in the MEOconstellation at one time to resolve a GPS 3D position. Fourpseudo-range observations and a clock bias parameter are included inGNSS signals to compute the X, Y, and Z (latitude, longitude andaltitude) coordinates of a point.

Indoor applications that require high levels of precision and accuracyfor location sensing can benefit from a convenient way to obtainsecondary or supplemental base-station distance readings, particularlysince time-of-day, poor weather conditions, and other factors caninterfere with the number of satellites and/or towers available forcommunication with the Smartphone's receivers at any given time. The USGPS system has been operational since 1995 and other countries are stilltrialing CDMA GNSSs, so satellite coverage remains a hurdle with just24+ MEO satellites at ˜12 mi altitude travelling at ˜7,000 mph. The U.S.government is pursuing a modernization effort known as GPS III, with oneaim to improve the accuracy and availability for all users, and newersatellites are planned for launch in 2013+. Targets for signal strengthimprovements of GPS III are 100 fold, and geocode location accuracycould see similar gains that are only expected to improve overtime withnewer GNSS capabilities likely coming online from international systemslike the EU's Galileo constellation. Even with these improvements, theneed for a localized DGPS supplemental reference location is anticipatedwell into the future for some consumer settings. The SDGPS method useslocal wireless devices to create base-station locations that can augmentthe DNSS satellite signals and the terrestrial-based DGPS readings.

System

FIG. 1 depicts an exemplary system for use with embodiments of theinvention. A mobile device 101 (smartphone or tablet) makes an internetprotocol connection with a wireless access point 104 Internet AccessGateway via Wi-Fi (e.g., 802.11). Mobile device 101 can include aprocessor, a screen and keyboard or touchscreen interface, a GPSreceiver for sensing GPS coordinates from GNSS signals, a plurality ofsensors for determining acceleration and orientation, including magneticsensors and accelerometers, a Wi-Fi transceiver and antenna, and acellular transceiver and antenna suitable to communicate with a CDMA,GPRS, GSM, LTE, or other carrier network. Wireless device 101 canestablish a cellular connection with a carrier's wireless tower 107 via3G or 4G wireless communications or the like. Wireless device 101 canreceive GPS signals from GNSS Satellites 106 via traditional signalingprotocols. Tower 107 can also receive GPS signals from GNSS Satellites106 via traditional signaling protocols to determine geocode or have apredetermined location recorded. Since the tower 107 is in a fixedlocation, extended stationary measurements that determine precise geolocation (Lat/Long/Alt) may be possible. Alternatively, tower 107 maynot need an active GPS receiver; a precision DGPS survey can be done tocreate a database of known tower locations. In some embodiments,wireless access point 104 can also make a Wi-Fi connection to a Wi-Fiantenna on tower 107. In some embodiments, mobile device 101 cancommunicate with servers on the Internet via broadband connections ofeither wireless access point 104 or tower 107 for accessing cloud-basedservices and for offloading computationally intensive processes. In someembodiments, wireless device 101 can communicate with other IP deviceson a home network that shares access to wireless access point 104.

Wireless device 101 determines DGPS precision geocode coordinates viacommunications between the GNSS satellites 106, one or more towerlocations 107, wireless access point 104 and/or other nearby Wi-Fidevices, such as Wi-Fi enabled set-top boxes. A user can calibrate thelocation of a video display 102 using the method described in FIG. 4.Video display 102 receives a physical connection (i.e., HDMI) from avideo set top box (STB) 103. Video STB 103 makes a Wi-Fi connection withwireless access point 104, and optionally communicates directly overWi-Fi or Bluetooth with mobile device 101. A media and video contentserver 105 makes a wired or wireless Ethernet connection to the VideoSTB 103 and the wireless access point 104. This could be a long distanceInternet connection of a LAN or connected storage connection. Media andvideo content server 105 can supply television signals and/or recordedprogramming and information that may be accessed by a user interactingwith video display 102.

The system in FIG. 1 uses position and orientation information gatheredfrom sensors (including antenna and receivers) on the mobile device todetermine where the mobile device points and communicates this to videodisplay 102. This may be accomplished in a number of ways in differentembodiments as described throughout. In some embodiments, the processoron the mobile device can be programmed to calculate its position andorientation with sufficient precision and communicate this position andorientation information to a server across a network. The server may bea remote server on the Internet, including a cloud-based server, or maybe software operating on local hardware, such as a set top box in theuser's home network. This information can be communicated across a Wi-Finetwork or a cellular network, including any additional transportnetworks, such as the Internet. In some embodiments, the mobile devicedoes not calculate its entire position and orientation. The wirelessdevice may gather position information, such as GPS coordinates,distance measurements relative to known locations, such as time offlight information or signal strength information relative to cellulartowers or Wi-Fi access points, or other wireless devices, such as a settop box, and communicate this position information to the server foradditional calculation. This may allow the server to use more robustprocessing components to calculate a more precise location. The servermay also have access to additional information, such as precise geocodesfor the devices with which the mobile device communicates. This mayoffload processing from the mobile device, allowing a smaller or lessprocessor intensive client component to operate on the mobile device.

It should further be appreciated that in some embodiments, the serverand set top box may be separate, while in other embodiments the servermay be software operating on the set-top box. In some embodiments, themobile device, the server, and the set-top box all participate indetermining the mobile device's position. Preprocessing tasks involvedin calculating this position may be broken down between the mobiledevice, the server, the set-top box in any reasonable manner evident topersons of ordinary skill.

Similarly, in some embodiments, the mobile device determines itsorientation using onboard sensors, such as accelerometers, magneticsensors, or by extrapolating its orientation by observing changes inposition, and communicates this orientation information to the serverand/or set top box. In some embodiments, the server and or set-top box103 may participate in calculating an orientation of the mobile devicefrom this orientation information.

Once the position and orientation of the mobile device has beendetermined, one of the mobile device, the server, or the set-top box 103(or any combination thereof) may then apply this position andorientation to an environmental model that describes the position andorientation of the TV display 102 to determine where on the screen themobile device is pointing. Video set-top box (STB) 103 can then displaya cursor or other visual feedback to the user about the extrapolatedposition on the screen that the mobile device points to. In someembodiments, no visual feedback is displayed, but a user may interactwith the screen and select icons or menu items by clicking or gesturingwhile pointing at a portion of the screen. Video STB 103 may communicatewith video display 102 to display any menus, cursors, or informationselected by the user.

Video STB 103 can record pointer/cursor click locations on video display102, and transmit the selection of information to media content server105 for playback of selected video content. When an optional Internetconnection is available between video display 102, and the wirelessaccess point 104, selection of OTT video assets that bypass video STB103 can also be played out on video display 102.

FIG. 2 depicts an exemplary software architecture for use with someembodiments of the present invention. The mobile device 101 containshardware/software modules 201 for determining position and orientationinformation. Exemplary hardware includes a magnetic compass, anaccelerometer, a level, cellular and Wi-Fi transceivers, and a GPSreceiver. Exemplary software includes APIs for accessing these hardwaremodules. In addition, one or more apps or the operating system caninclude software modules for accessing and calculating position fromGPS, DGPS, SDGPS, and receiving user input from a user-interface (e.g.,a pointer API). These modules can interface to a thin client mobile appvia standard APIs to convey this information to one or more servers orset top boxes. Mobile device 101 can communicate via two-waycommunication with client modules on video STB 103 or video monitor 102.These clients can include IPTV thin client 202 operating on videodisplay 102, and/or a STB thin client operating on video STB 103, and/ornative STB client application. In some embodiments, software operatingon video STB 103 can also be described as a server. In some embodiments,IPTV client 202 or an STB client 203 acts as clients for backendservices 205, which may be provided remotely via Internet communicationwith a cable provider.

In some embodiments, mobile device 101 may also have two-waycommunication with an optional media server and/or gateway application203, by proxy of STB software 202. In some embodiments, mobile device101 may also have two-way communication with a cloud server and/orheadend content services 204, which host video content for delivery to adisplay device via STB, media server/gateway, or embedded SmartTVapplications, by proxy of STB software 202. Backend Services 205 cancontrol the rights and privileges of the other components' interactionswith each other. These services include management, billing,authentication, identity management, entitlements, personal settings;e.g., parental controls, and digital rights management (DRM).

Calibration and Initialization

FIG. 3 illustrates the operation of software instructions used toimplement embodiments of the present invention. The steps shown in FIG.3 may be carried out by an app on mobile device 101, server software aremote server or a set-top box, or any combination thereof. At step 301,client software 201 (described as a pointer app) initializes. This caninclude gathering any computational, communication, and sensor resourceson the mobile device necessary for operating the client. At step 302,the user selects a pre-calibrated display or chooses to identify a newdisplay. If no new display is selected, the pointer app retrievesprecision geocode coordinates for the mobile device via methodsdescribed herein using onboard sensors at step 303. At step 304, theclient determines whether it recognizes stored coordinates for a nearbydisplay.

If the pointer app determines that a display is nearby, the pointer appcan begin facilitating the interaction of mobile device in the display.At step 305, the pointer app engages the accelerometer on the mobiledevice to begin sensing the motion of the device. At step 306, themotion of the device is monitored to sense gestures performed by theuser. Examples of gestures may include, for example, swiping in eitherdirection, a quick flick/jab to indicate a selection, rotating thedevice to operate a jog dial, tilting the device in any direction, whichmay indicate that the user would like to move the cursor on the display.Gesture interpretation can also make use of embedded position sensingmodules via provided APIs, at step 307. The pointer app may access theseAPIs to determine direction, yaw, tilt, etc. At step 307, the directioninformation provided by the sensors can be used to determine not onlygestures, but also the location on the display that the mobile devicepoints to.

At step 308, the pointer app may render a pointer location on thedisplay. This may include highlighting an icon that is selected or byproviding a movable cursor on the screen.

If no display is recognized near the device at step 304, at step 309,the software determines whether a partial match can be found for anearby display. If no match can be found, the method returns to step302. If at least a partial match can be found, at step 310, arecalibration sequence can begin. An example of this process is shownwith respect to FIG. 4. Recalibration process 310 may be likecalibration process 400, shown in FIG. 4. To calibrate the screen, themobile device is moved to predetermined locations on the display inthese locations, allowing the pointer app to record the positions inphysical space using the GPS and supplemental GPS positioning describedherein. At step 311, the recalibrated positions of the display can beused to address any drift in the DGPS positioning of the mobile device.For example, if a TV display has not moved and the recalibration step310 indicates that the model represents the display as having moved bysome distance, that distance can be used to offset the determinedposition of the mobile device. This can allow the mobile device'sabsolute position to be estimated using less accurate GPS signals, whilethe relative motion of the device can be used to approximate the devicescurrent location with increased accuracy.

If DGPS/SDGPS precision geocode values determined sensed by the mobiledevice are corrupted or drift has been detected such as due to signalnoise, a pointer tuning and optimization sequence 312 can be used. Insome embodiments, optimization sequence 312 can utilize manual inputsfrom the user, such as by allowing the user to tell the system that themobile device is packed a known location, such as a corner of a coffeetable. Accelerometers and orientation sensors in the mobile device canthen be used to determine the location and orientation offset from thatknown position as user begins to use the mobile device.

To add a new display, at step 320, the pointer app begins a newcalibration sequence. The sequence can be used to add a new display ornew environment. At step 321, GPS-based coordinates are gathered fromthe mobile device when the device is placed at a known location in theroom with the display. For example, the device may be placed on a tablenear where the mobile device will be commonly used as a pointer devicefor interacting with the display. In some embodiments, GPS coordinatescan be gathered for a predetermined period of time, such as an hourduring a calibration phase, allowing coordinates to be accumulated andaveraged. This may increase the accuracy in calibrating the location ofthat position in the room. In some embodiments, the position used atstep 321 may be used as an initialization position for using the mobiledevice as a remote pointer in the future. For example, a device may beplaced at a known location at the beginning of each use of the device asa virtual pointer. In some embodiments, a user may be provided with asticker having a QR code or other optical marks or RFID tag, allowingthe user to easily initialize the mobile device as a virtual pointer ata known location during each session she uses the mobile device as avirtual pointer. The device may be placed at or near this locationduring future uses, allowing position sensors and accelerometers to beused to determine or improve the calculation of relative offsetpositions from this initialization position as the user uses the mobiledevice as a virtual pointer.

During initialization phase, at step 322, the mobile device determineswhether or not a geocode with a desired precision is available for themobile device. This geocode may be provided by GPS receivers in themobile device. In some embodiments, a mobile device may be placed at theinitialization position for one or more hours to accumulate an averagegeocode from GPS signals to be used as the baseline position. In someembodiments, additional signals can be used to provide a secondarydifferential GPS position. For example, Wi-Fi signals or other signalsfrom networking devices at fixed locations, such as Wi-Fi hotspots andcellular towers can be used to provide additional positioninginformation to supplement GPS signals. In some embodiments, time offlight between the mobile device and the other networking devices isused to approximate the relative distance between mobile device andthose other networking devices. In some embodiments, signal strength ofthe other devices may be used to assist in determining the distancebetween mobile device and those fixed location networking devices.

If a precise geocode is available, at step 324, the user goes through acalibration sequence described in FIG. 4 to identify certain locationson the display and records these. The pointer app then calculates theGPS locations for these locations on the screen. These DGPS positionscan be determined using any of the methods described herein, includingrecording GPS positions, as well as using supplemental positioninformation, including recorded offsets from the fixed location in step321 using the accelerometers, or by utilizing calculated distances fromother networking devices, such as Wi-Fi access points and cellulartowers.

At step 325, if no precise geocodes are available, the pointer appdetermines whether GPS signals are sufficient to allow increasedaccuracy by recording a geocode over a longer period of time, such asone or more hours. If the option is available, the user can beinstructed to place the device at a fixed location in the room and leaveit for a predetermined amount of time until a sufficient GPS lock can beobtained. During this initialization, GPS signals, as well as signalsfrom other networking devices, can be used to refine the geocodelocation of the mobile device. Once a sufficient geocode location can berecorded for the location of the mobile device, the user can proceed tocalibration step 324, where the user will record certain locations ondisplay, such as each corner of the display. This allows the pointer appto determine a relative location of portions of the screen relative to aknown location of the room to allow the system to utilize anenvironmental model of the display relative to the room when the userattempts to interact with the display a later time. Once step 324 iscomplete, the method can return to step 303.

If no delayed geocode is available at step 325 (or if drift has beendetected at step 311), the pointer app proceeds to step 326, whereby theapp starts a secondary DGPS calibration sequence. This sequence isdescribed with respect to FIGS. 4 and 6. The method then returns to step303.

FIG. 4 illustrates the sequence of steps 400 that a user can be walkedthrough to perform calibration of locations in the physical space inwhich the mobile device and pointer app will be used. At step 401, auser stands at a predetermined location in a room, such as the center ofthe room. The predetermined location used may be at the preference of auser. For example, in some embodiments, a user may select a corner of atable in a room or any position that may be close to where the user willoperate the remote app. During step 401, on device 101 uses a remote appto determine an estimate of a geocode for the predetermined location. Insome embodiments, this position will be determined with a minimumprecision necessary to give a user a satisfactory user experience whenusing remote app in the future.

In some embodiments, an accurate geocoding of the predetermined locationmay not be necessary. For example, in embodiments where a mobile devicewill initialize at that predetermined location each time the remote appis used (for example, requiring the user to check-in his mobile deviceat a predetermined location on a coffee table when he initializes theremote app to interact with a display each time), the geocode for thatlocation can be approximate. In these embodiments, once the mobiledevices check in with that predetermined location during future uses,only the relative position of that device to that predetermined locationneed be used to provide fairly precise calculation of the position ofthe mobile device in the room to interact with video display 102 in asatisfactory manner. The geocode of the predetermined location isrecorded by the remote app as a result of step 401.

At step 402, the user holds mobile device 101 at the predeterminedlocation and orients the device and a level position pointing toward thecenter of the display 102. This orientation information can be recordedby the remote app to provide a baseline orientation for mobile device101. This baseline orientation may be obtained from various orientationsensors in mobile device 101, described throughout. In some embodiments,the steps for a 402a can be repeated not only during a calibration phase400, but during a viewing phase, each time the user enables the remoteapp. It should be appreciated, that in some embodiments, repeating steps401 and 402 may not be necessary where accurate position orientationinformation can be obtained using the sensors aboard mobile device 101.

At step 403, during a calibration phase, the user walks to video display102 and holds mobile device 101 at predetermined locations on thedisplay, such as each corner of the display. This allows the remote appto record the position and orientation of the video display. It can alsobe used to determine information such as the size of video display 102,allowing the remote app to determine when a user is pointing mobiledevice 101 at the display 102 and to determine what portion of display102 the user is pointing at. For example, a user may be instructed tohold at the upper left corner of the display. User may click a prompt onmobile device 101 when he has positioned the device at the requestedlocation. The user may then be walked through a sequence of steps toplace the mobile device at different corners of the display, allowingthe remote app to record these positions as geocodes or positionsrelative to the predetermined location recorded in step 401.

In some embodiments, the user may be instructed to record variouslocations throughout the room where the user is likely to use remoteapp. For example, in step 404, a user is instructed to sit at apreferred viewing location and hold the mobile device as he normallywould while using the remote app. When the user sits as instructed andholds mobile device 101 in a comfortable position that she will useduring normal viewing, the remote app may record this position as ageocode or position relative to the predetermined location from step401. The remote app may also record the orientation of mobile device 101when the user points the device to the center of video display 102 atthis new location. Step 404 may be repeated for various preferredviewing positions.

In addition to pointing to the center of video display 102 to calibratevarious viewing positions, at step 405 the user may also be instructedto point mobile device 101 at various positions on the video display tofurther refine the expected orientation for a mobile device at theseating location when interacting with the video display. For example,after a user has calibrated the mobile devices position at a seatinglocation at step 404, the user may be instructed to point the devicefrom that location to various positions on the video display, such aseach corner of the video display. In some embodiments, the user may beasked to refine the orientation of the device from each seatingposition. At optional step 406, once a user has pointed a mobile devicegenerally to the corners of via display, the user may be instructed topoint the mobile device at smaller objects on the screen, such as iconsor targets. Step 406 may be useful for further refining the precision inthe orientation of mobile device 101 when interacting with displayswhere further granularity may be desired.

At step 407, the user may then begin to use the mobile device 101 is avirtual pointer. Mobile device 101 receives signals from GPS, DGPS, andSDGPS sources, such as Wi-Fi hotspots and cell towers.

Operation

FIG. 5 illustrates the system components and user interfaces that may beused to establish virtual pointer connections between a mobile device101 and video screen 102. Video screen 102 may include physicalconnections, such as HDMI connections, to set-top box 13 and mediaserver and gateway 105. As discussed with respect to FIG. 2, thesecomponents may include software portions 202 and 203. Setup boxapplication 202 delivers requested content to video display 102 fromeither a Media Server 105 or a remote cloud-based content hostingservice 204 once authorized by a remote cloud-based BSS/OSS 205. Avirtual pointer connection (e.g., providing motion sensing, virtualpointer, gesture recognition, and wireless mouse functionality) isestablished via physical proximity between mobile device 101 and videodisplay 102. The interpretation of this virtual physical connection canenable a user to interact with video display 102, as if the mobiledevice was directly linked to the TV like an interactive laser pointer.In addition, the user may interact with video display 102 using gesturesand the screen of mobile device 101. An overall user interface may allowmotion sensing, point and click interaction, and gesture sensing usingthe motion sensors or the touchpad of mobile device 101.

In some embodiments, the display 102 and the display on handheld device101 display duplicate interfaces. For example, both displays may displaya copy of the TV Guide, allowing a user to select programs from theschedule and navigate between channels. In some embodiments the display102 and mobile device 101 simultaneously display replicated contentnavigation icons. In some embodiments mobile device 101 may allow a userto interact with the touchscreen to provide a traditional remoteinterface, including common thumb-based remote buttons that may beselected. Exemplary commands that may be available via the touchscreeninclude, but are not limited to, click okay, up, down, left, right, lastnavigation, save/record, remind, add to favorites, share, comment, anddelete/block. In some embodiments, the user may decide whether tointeract with video display 102 via the virtual pointer interface or viathe touchscreen interface of mobile device 101.

Virtual pointer connection between mobile device 101 and screen 12 mayalso allow touchscreen-based scrolling, allowing a user to flick betweenmenus using the touchscreen while pointing the device at the TV screen.For example, a user may scroll through a library of graphic icons usingthe touch screen of mobile device 101, while the system simultaneouslydisplays information or selected content on video display 102. In someembodiments, the content may be synchronized, such that as the userscrolls through items on the touchscreen, information on the display 102updates in real-time in response to the user's interaction with thetouchscreen. In some embodiments, this may include replicating contentnavigation items presented on both the TV monitor display and the mobiledevice touchscreen to enhance user interaction using a virtual pointerconnection.

In some embodiments, dual screen functionality may be utilized, allowingthe user to simultaneously interact with a menu on video display 102,while also interacting with user-interface on the display of the mobiledevice, which may be different from the menu and information displayedon the video display 102. In some embodiments, for example, the touchscreen on mobile device 101 may display traditional arrow-basednavigation icons corresponding to traditional buttons on a remote, whilethe user may simultaneously interact with the video screen by motioningthe mobile device up, down, left, or right to achieve similarnavigational results. This may present a more intuitive user experience,allowing the user to choose how to interact with video content whileusing his mobile device. In this manner, in some embodiments, thevirtual pointer capabilities allow the mobile device to actsimultaneously has a remote or a virtual mouse.

FIG. 6 shows an exemplary communication diagram illustrating thecommunications between a mobile app 201 on mobile device 101, a set-topbox client 202, and a content navigation server 204. Content navigationserver 204 works with mobile app 201 to determine how the user intendsto interact with the display that he points at. Setup box client 202 andcontent navigation server 204 can also communicate with other services,such as content hosting services 203 and backend content managementservices 205. It should be appreciated that, in some embodiments,content navigation server 204 and set-top box client 202 may be softwareoperating on a common set-top box. In some embodiments, contentnavigation server 204 may be a cloud-based server that may be balancedacross multiple server computers that communicate with mobile app 201and set-top box client 202 via the Internet.

At step 420 mobile app 201 initiates the remote pointer application andcommunicates to STB client 202 that the remote pointer app has beeninitiated. At step 421, mobile app 201 determines a location for themobile device, such as gathering a geocode from GPS, DGPS, and/or SDGPSsignals. At step 423, mobile app 201 sensors communicate with server 204to inform the server of the mobile devices location and orientation.Information transmitted in step 423 may include any reasonable form ofposition and orientation information about the mobile device, such asgeocode information, yaw, pitch and direction information, distancemeasurements from known locations, etc. At step 425, server 204 maycommunicate with backend content management services 205 to inform it ofknown locations of devices described by the location and orientationinformation received from mobile device and to verify the displayauthorizations for the user of the mobile device. In some embodiments,content navigation server 204 will also verify the controller locationsat step 426 by communicating an acknowledgment to mobile app 201.

At step 430, mobile app 201 refines its position using SDGPS signals.Mobile app 201 may communicate with a terrestrial DGPS tower or furtheraugment SDGPS geocode resolution through communications with other localwireless (such as a WAP or cell tower) to refine its precision geocodelocation. When real-time or near-real-time DGPS precision geocodecalculations are processor intensive beyond the capabilities of themobile device, they can be off loaded to the content navigation server204 by sending unrefined position and orientation information frommobile device to server, allowing server to calculate position andorientation of the mobile device (step 431). The server may then send(step 432) controller location data about the mobile device to STBclient 202. This may include sending a list of object locations fromcontent navigation server 204 to set-top box client 202 and receivingacknowledgment. At step 434, the server and mobile app can run alignmentscripts between the known display location's precision geocodes and thehandheld controller's dynamic DGPS updates, allowing the server to trackwhat the users pointing at and to convey this information to the mobiledevice to allow the mobile device to update any touchscreen displays. Insome embodiments, content navigation server 204 correlates the locationand orientation information received from the mobile device with anenvironmental model that includes known locations of objects in theenvironment.

At step 436, content navigation server 204 can inform STB client 202 ofthe current portion of the screen that the mobile device points to. Thiscan include one or more clicks where the set-top box displays selectablecontent. This information may be used to understand that the user hasselected a button or icon when the user makes a gesture or clicks histouchscreen. At step 437, the user begins interacting with content onthe display. Mobile app 201 gets motion and gesture inputs from itson-board embedded accelerometer, level, and compass modules and STBclient 202 detects that a TV pointer sequence is ready. At step 439, app201 sends display point plane coordinates to navigation server 204 toinform the server when a user has interacted with the touchscreen, suchas clicking an. At step 440, navigation server 204 sends clickablescreen hot zone information to the STB client 202 for rendering clickzones on the TV display. At step 441, Mobile app 201 sends pointingcoordinate to STB client 202 and navigation server 204. Navigationserver 204 sends click zone instructions to the STB client 202. Mobileapp 201 sends input/output commands to the STB client 202.

At step 443, in response to user selection using virtual pointer, STBclient 202 communicates with backend content management services 205 torequest content authorization for the content corresponding to theselection of the user. At step 445, mobile app 201 and STB client 202exchange pointer clicks on information and continue interacting.

FIG. 7 depicts the interaction between a mobile device and varioustransmitters that may be used to provide positioning information. When amobile device initializes the remote control pointer application, theapplication requests that the STB client makes ready a remote pointer onthe video display. The app then receives input and output commands fromthe user of the device. In order to provide an interactive virtualpointer experience, the mobile device 101 receives geocode updates fromits onboard GPS sensors communicating with a constellation of GNSSsatellites 450. This can provide a rough estimate of the position ofmobile device. In some embodiments, further precision is needed toprovide meaningful user experience. To provide greater precision, mobiledevice 101 communicates with a plurality of networking devices. This caninclude cellular tower 107, cellular tower 451, and cellular tower 452.These towers may use any conventional cellular communication protocol.The geocode coordinates of these fixed location towers can beestablished by publication from the cellular carriers operating antennaeon the towers. Mobile device 101 can enhance its position information byestimating distances from these towers using TDOA (Time Difference ofArrival) and AOA (Angle of Arrival) radio frequency techniques. Thesedistance measurements can be made by observing a time of flight ofasynchronous signals, comparisons of arrival times of synchronizedsignals, observing angle of flight of received signals, comparisons ofsignal strength, etc.

For example, a cellular carrier may utilize its cellular towers to offera value-added service, whereby the towers each broadcast a commonsynchronized signal at a synchronized time interval. Devices receivingthis signal can compare the arrival times from the various towers,allowing the device to triangulate its location relative to plurality ofcellular towers making this information useful to enhance or replace GPSinformation, and may be suitable for providing enough precision in thelocation of the estimated position of mobile device 101 to provide ameaningful user experience.

In addition, other one networking devices, such as set-top boxes withwireless antenna, and such as 802.11 wireless devices and access points453, can be used to provide fixed location beacons for mobile device 101to measure or estimate its position. For example, 802.11 wirelessdevices can use TDOA in the way cellular communications networks do, sotime of flight can be used to measure the distance between mobile device101 and access points 453. Similarly, pinging a wireless router can beuseful in estimating the time of flight of signals between the mobiledevice and the router. Any latency in the router can be characterized byregular pings of the router. Once latency is estimated for the wirelessrouter, the remaining time interval between sending and receiving theping at the mobile device can be estimated as the time of flight (roundtrip) between the mobile device and a wireless router. This can be usedto provide a distance estimate to assist in improving the accuracy oftriangulation calculations for estimating the position of the wirelessdevice.

Due to the distance of GPS satellites (Medium Earth Orbit between˜19,000 km and ˜23,000 km) and interference to the GPS signals in space,corrections can be done to improve geo-coding accuracy using a GBAS(Ground Based Augmentation System). The US FAAs WAAS (Wide AreaAugmentation System) specification, for example, provides a positionaccuracy of 7.6 meters (25 ft.) or better (for both lateral and verticalmeasurements), at least 95% of the time. Actual performance measurementsof the system at specific locations have shown it typically providesbetter than 1.0 meter (3 ft., 3 in) laterally and 1.5 meters (4 ft., 11in) vertically throughout most of the contiguous United States and largeparts of Canada and Alaska. In Europe, the European Space Agency wentoperational with EGNOS in 2009 to augment the Glonass and Galileo GNSSsatellite systems. A similar system in operation is the USCG's andUSDOTs NDGPS (National Differential GPS) which has over 85 DGPStransmitter sites in operation and plans for deploying over 125. As of2012, it is estimated that there are over 400 terrestrial DGPS beaconsin service globally. Commercial systems include StarFire and OmniSTAR,with John Deere's SF2 introduced in 2004 achieving decimeter accuracy95% of the time using 25 ground reference stations worldwide. John Deerealso offers a StarFire® real-time kinematic DGPS tripod for use inagricultural settings with centimeter location accuracy.

The US DoD has estimated that DGPS accuracy degrades 1 meter for every150 km of distance to the broadcast site. Radio propagation is furtherdegraded by indoor transmission, so high-resolution geo-location toseveral centimeters of precision may be assisted with differentialbeacon and/or distance measurements to known fixed terrestrial radiotransmitter/receiver locations within 5 km. Most GSM and CDMA cellulartowers are capable of propagating signals at longer ranges but areconfigured to disperse signals to less than 10 km. In populated areascellular towers are often spaced 2-4 km from each other. Public Wi-Fiaccess points are typically clustered closer together than cellulartowers with the 802.11n outdoor range being 250 m, however the802.11y-2008 amendment to the IEEE 802.11-2007 standard allows for useof the higher power 3,650 MHz band now being used by some wireless andbroadband providers to extend Wi-Fi range to 5 km from the Wi-Fi accesspoint. Many broadband providers now operate several thousand publicWi-Fi access points in individual metropolitan areas providing thecoverage to enable using those locations for augmented GPS reference andgreater high resolutions precision augmented GPS (AGPS) and DGPS geocodereadings. 802.11ac, also known as 5G Wi-Fi, is the latest draft of theWi-Fi IEEE standard, and in 2012 Broadcom and others introduced thefirst SoCs for new devices which can operate in the 5 GHz band. 5G Wi-Fiincreases range by ˜20% while increasing data rate speeds overthreefold. Broadcom has estimated that, by 2015, the number of globalpublic Wi-Fi hotspots will be approaching 6 million, allowing forcellular-like roaming for Wi-Fi enabled devices. In addition to publicWi-Fi hotspots the number of private Wi-Fi access points that could beconfigured as AGPS locations is already in the 10's-of-millions in mostdeveloped regions around the world.

In some embodiments, to achieve the geo-positioning resolution in thehandheld device to establish the desired motion pointing and gesturesensing interface with a video screen or smart building device, the GPSsignal should be augmented with traditional AGPS and DGPS methods, andthen refined through a SDGPS process by communication with either anearby DGPS beacon or a plurality of nearby known fixed terrestrialwireless transmitter /receiver (e.g., GSM, CDMA, UMTS/HSPA, 3GPP orWi-Fi) locations with published precision geo-code centroid informationaccessible to the handheld. In some cases the SDGPS process and befurther refined with a Real-Time Serial Multilateration (RTSM)calculation process that would culminate in replacement of the originalGPS reference where four or more AGPS and/or SDGPS sites are in range.In addition, common syntax for GPS geocodes can be modified to enablereal-time processing of AGPS, DGPS, and SDGPS calculations. In someembodiments, the new syntax can be used while offloading processing ofthose calculations to cloud-based servers. Commonly used syntax for$GPGGA (fix data) lat./long=dddmm mmmm, antennae altitude=0, M, [or] 0,f, and geoidal height=0.0, M, and for $GPGLL (geographic position)latitude=ddmm.mm, longitude=dddmm.mm. Common degree-minute conversionsin database systems produce decimal-degree values (ddd.ddddd) which aretoo coarse for high resolution precision location measurements. Latitudeand Longitude can be first converted to a common Degree-decimal-minuteformat with a minimum six decimal integers for computation, andelevation can be converted to meters in the 0.000 format with a minimumof three decimal integers. Several centimeters will be roughlyequivalent to 0.000016 minutes depending on the latitudes where themeasurements are being taken. Because altitude above mean sea level andheight from geoid centroid are inherently less accurate measurementsthan lat./long, it is recommended that elevation be calculated atmillimeter resolution for differential positioning corrections to heightof the handheld device during indoor operation.

FIG. 8 illustrates an example of refining the location of mobile device101 for use with some embodiments. At start up, a mobile app accessesGPS modules in the device to get geo-location coordinates from aninitial fix on GPS signals provided by GNSS constellation 106. Thisfirst fix can be accelerated using aGPS (assisted GPS) data sets storedin a cloud or on the mobile device in persistent memory. This canproduce a geo-location accuracy of several meters horizontal and tens ofmeters vertical. The accuracy range of GPS is illustrated by ellipticalsphere 461.

The mobile app requests AGPS (augmented geocode values using GBAStechniques) geocode updates from the device's onboard embedded GPSmodule communicating with the GNSS satellite constellation andcommercial DGPS systems. This second step of geo-code refinement mayproduce a refined geo-location, in some embodiments, producing anaccuracy of +/−1 meter horizontal and several of meters vertical(refined elliptical sphere 462) with minimal processing.

Next, the mobile app identifies cellular towers (107) in range andwireless access points (453) in range, and determines radio frequencytransmission time differential measurements between the handheldantennae of the mobile device and each transmitting radio antennae inrange (of the towers and access points 107 and 453) and ranks theclosest antennae with the objective of logging four or more wireless orcellular access points within a threshold distance, such as 4 km. Themobile app searches for active DGPS beacons. If any are, within asatisfactory distance range for precision calculation of high resolutiongeocodes, the SDGPS process is invoked, which may leverage cloud-basedcalculation processing, to produce high-resolution precision geo-codesfor the device in real-time. An RTSM technique, such as shown in FIG.15, can also be used to further refine the high resolution localizationof the mobile device. These higher resolution geocodes can berepresented by further refined elliptical sphere 463.

If no DGPS beacon is in the desired range (˜300 km using) the mobile appthen requests information from the closest transmission sites toindicate if any are registered passive high-resolution DGPS referencesites with accessible known location data. If any are, the SDGPS processis invoked, which may leverage cloud-based calculation processing, toproduce high-resolution precision geo-codes for the device in real-time.

If none of the transmission sites in range are registered passivehigh-resolution DGPS sites with accessible known location data, theSDGPS process utilizes non-precise location data to producehigh-resolution geo-codes for the device in real-time, which can bemanually verified by the user for usability. For example, the mobile appmay request that the user place the mobile device at a known location,such as a corner of a coffee table. The geocode calculated using theSGPS process can be compared to that known location and an offsetapplied to correct any error. It should be appreciated that manualverification may not be required each time user starts up the mobileapp, but may be helpful when sufficient DGPS signals are available.

In instances where the non-precision real-time high-resolution geo-codescalculated for the device and/or by the Mobile App on the device areinsufficiently accurate to provide a desired user experience (such as+/−5 cm, in some environments), a nearby Wi-Fi access point can beconfigured as an active DGPS beacon, or more than one nearby Wi-Fiaccess points can be configured as passive high-resolution DGPSreference sites. Furthermore, in buildings with poor radio propagation,local active DGPS beacons may be added.

FIG. 9 depicts an exemplary system 470 that allows a user to utilize thevirtual pointer system to provide interactive functionality withinteractive objects in the environment. These can include non-displaydevices, such as home automation devices or security systems. Otherinteractive devices will be readily apparent to one of ordinary skill inthe art. Interaction with these interactive devices may be in additionto interacting with a video display. For example, a user may have asmart home that includes a flat screen TV and many automated systems,such as environmental controls, lights, security panels, etc. In someembodiments, interactive devices may utilize Z-wave protocols tocommunicate with home automation servers.

One or more mobile devices 101 can utilize an application to establish avirtual pointer connection with video monitor 102, as well asconnections with interactive devices, including light bulb 474 that maybe controlled by a light switch, light switch 475, and security andautomation panel 476. The virtual pointer connections can be establishedusing any of the methods described herein.

An app on one of the mobile devices 101 can access position orientationinformation from onboard motion, position, and location sensors, e.g.,accelerometer, magnetometer, gyroscopic sensor, and GPS, for accessingposition orientation, location, and motion information relative to themobile device. The app then initializes and accesses location data fromthe GPS module's initial first fix obtained by measuring radio signaltime of flight distance to multiple GPS GNSS satellites 450 fortrilateration. Assisted GPS (aGPS) location data can be accessed from aremote datastore 473 to speed time to first fix and improve accuracy andresolution of first fix geocodes (latitude/longitude/altitude). The appthen identifies the closest cellular tower radio antennae (107) andwireless data access points (453) (e.g., network devices). Foradditional refinement and high-resolution calculation of locationinformation using trilateration based on radio signal time difference ofarrival (TDOA) and further geo-location refinement using triangulationbased on angle of arrival (AOA) of radio signals exchanged betweendevice 101 and these network devices. The mobile app then accessesaugmented ground based GPS (GBAS) location data and additionalDifferential GPS (DGPS) processing resources to calculate highlyaccurate high-resolution geocodes for mobile device 101. An appoperating on the mobile device can utilize server resources 204 and 205as described throughout.

A mobile app on the mobile device 101 then identifies nearby interactivedevices, such as television screen 102 or interactive devices 474, 475,or 476 that are pre-registered with known high-resolution geocodelocations for pointing and motion sensing interaction using thecalibration techniques described throughout. The mobile app can also beused to register accurate high-resolution geocode locations for newinteractive devices with unknown locations.

FIG. 10 depicts a flowchart of the operation of a remote pointer systemthat utilizes the refinements depicted in FIG. 8. Method 500 includesthe initial steps that can be used when a user first uses the mobiledevice including any calibration of the device and of the environmentalmodel. At step 502, a mobile app on a mobile device initializes when thedevice is turned on or application is selected. At step 504, the mobileapp accesses assisted GPS (aGPS) data stores, which may be provided bycloud-based services. At step 506, the mobile app gets GPS coordinatesusing the mobile devices GPS receiver. This may be derived from GNSSsignals. At step 508, mobile app calculates a geolocation for the mobiledevice from the GPS coordinates received from the GPS receiver. In someembodiments, this step may be performed using a cloud-based service tooffload the mobile devices processor.

Once a rough GPS approximation of the position of the device has beencalculated, at step 510, the mobile app can access a database of towerlocations. These tower locations may be provided by third-party serviceor by the carriers that operate the towers. This database may be aremote database stored in the cloud. Using the mobile device's antenna,at step 512, nearby cellular towers can be detected. The location ofthese towers in physical space may be retrieved using the database. Thedatabase of tower locations may also include capabilities of the towers,including an identification of whether or not each tower has beenregistered as a high resolution geocode reference site or enabled foractive GPS beacon capability. At step 514, software determines which ofthe nearby cellular towers has DGPS capability. This determination mayalso be made based on signals broadcast by each the GPS beacon.

If a sufficient number of nearby towers are registered as DGPS referencesites, the app may proceed to step 516 where it determines the locationsof these the GPS beacons. This may be encoded in the DGPS signals ormaybe accessible in a remote database. At step 518, the DGPS locationsand received signals can be used to calculate the GPS coordinates thatmay have higher resolution or accuracy than the assisted GPS coordinatesdetermined from satellites.

If there are an insufficient number of nearby towers that are registeredas DGPS reference sites, a mobile app may still use the locations ofthese cellular towers to provide augmented GPS (AGPS) coordinates. Forexample, TDOA distance calculations can be utilized to determine aposition relative to a plurality of towers. Typically, AGPS coordinatesrequire having access to at least four tower locations. At step 522, theapp determines whether or not it has a sufficient number of towersproviding signals used by an AGPS algorithm. If so, at step 524, AGPScoordinates are calculated. If not, or if there are a sufficient numberof wireless access points nearby that may be used to provide refined aGPS position information, at step 526, the mobile app identifies thewireless access points that are in range. This can utilize the Wi-Fiantenna of the mobile device. At step 528, the locations of the detectedwireless access points can be retrieved from a data store which may bepart of the mobile app or maybe provided by cloud-based service, or acombination of the two. For example, a user may register the location ofhis home Wi-Fi router with his mobile app. Similarly, service providersmay provide wireless access points that may be accessed by customers.Service providers may record positions of these access points to beshared with a data store for the app to lookup the locations of wirelessaccess points. Using this wireless access point location information, aswell as signals received from the wireless access points, AGPScoordinates can be calculated at step 524.

Once DGPS and/or a GPS coordinates are calculated, method 500 proceedsto step 403. At step 403, a user calibrates the location of interactiveobjects, such as a display, as shown in FIG. 4. It should be appreciatedthat the interactive object need not be a display. Whereas a display mayhave multiple points to indicate the orientation and extent of planersurfaces, interactive objects/devices such as lights, light switches,thermostats, security panels, automation panels, etc. that may be IPenabled or in communication with a home automation server, may also becalibrated. These interactive objects may be defined by a single pointin space, such as the location on a wall where a switch or panel ismounted or the location in a room where a lamp is located. Onceinteractive devices have been defined in step 403, in some embodiments,method 500 proceeds to step 404. At step 404, a user may define preset,commonly used positions in the room to assist the mobile app in learningthose positions and the orientations for the mobile device wheninteracting with one or more interactive devices. Step 404 is describedwith respect to FIG. 4.

Once the GPS coordinates for the device are calculated and interactivedevices defined in calibrated in the physical space, method 500 proceedsto calculate SDGPS coordinates, as explained throughout. FIG. 15provides exemplary details for a method for calculating SDGPScoordinates using RTSM. At step 532, mobile app on a mobile deviceaccesses onboard motion and position sensor data to begin trackingmotion and determining the orientation of the device. Once the app hasaccess to the sensors, the method proceeds to step 405 as explained inFIG. 4. The user is instructed to point the mobile device at variouslocations in the room, including the positions of interactive devices.This allows the system to calibrate the positions in orientations of themobile device with respect to these interactive devices. At step 534,the display of the mobile device displays a user interface that reflectsthe type of interactive device that it is pointing at. For example, if amobile device is pointing at a video display, the set top box connectedto the video display may display a cursor on the video display thattracks the movement of the mobile device or provides guidance to theuser to guide him through the calibration process. In some embodiments,additional information may also be displayed on the touch screen of themobile device. When the user points the mobile device at a non-displayinteractive object, such as a security panel, a user interface isdisplayed on the mobile devices touchscreen. This may include providinga copy of the keypad on the device, allowing the user to punch in asecurity code.

At step 536, the app on the mobile device determines if the device isnow calibrated and is ready to be used as a virtual pointer. If so,method 500 proceeds to step 540, where the user begins using the deviceas a virtual pointer. In some embodiments, during future uses, thesystem may skip steps 403, 404, 405 and 532 through 534, as these stepsmay be most useful during a calibration and setup phase. If the systemis not sufficiently calibrated to be used as a virtual pointer, method500 proceeds to step 542. At step 542, software attempts to calibratethe device to a new DGPS device location. In some embodiments, this maybe done by instructing the user to place the device at a known location,such as a corner of a coffee table or at the location of a predeterminedsticker or RFID tag to restore the DGPS coordinates of the device to aknown location. From there, the device may have improved accuracy andprecision in positional placement by using other sensors, such asaccelerometers and compass sensors to assist the GPS system indetermining a relative offset position from the baseline location. Thismay be helpful where interference or indoor obstructions preventachieving an accurate GPS signal.

At step 544, the mobile app proceeds to process the RTSM positionalgorithm to track and improve the positional accuracy of the device.From there, method 500 proceeds to a recalibration phase whereby steps403 and 405 may be repeated to further calibrate the location ofinteractive devices. From there, the method repeats steps 534 and 536 toprovide a user interface to the user and to verify pointer functionalitybefore commencing gesture and pointer interaction at step 540.

FIG. 11 illustrates the usage case where mobile device 101 can be usedin an indoor environment to interact with a video display and otherinteractive devices. Device 101 can be located into spheres 801, 802 and803, as described with respect to FIG. 8. In addition, mobile device 101can use additional Wi-Fi localization details to maintain itshigh-resolution geocode reference, correcting for interior radio signalinterference and loss of GPS or DGPS signals. A mobile app on mobiledevice 101 can measure time difference of arrival (TDOA) of the signalsreceived from nearby wireless access points 104. Distance measurementsmay also include measuring a time of flight for ping responses sentto/from these wireless devices, or other wireless devices in thenetwork, such as wireless set-top boxes (such as 103), or may includemonitoring signal strength of these access points (e.g., reading Wi-FiRSSI signal strength information), which can be used to measure distancebased on Wi-Fi signals' known loss characteristics. In some embodiments,a learning algorithm on mobile device 101 can learn of the signal losscharacteristics of the Wi-Fi devices in an indoor environment the more auser uses the mobile device in that room, by comparing other locationcalculations, such as accumulating accelerometer readings to measurerelative distances. In some embodiments, multiple indoor wireless accesspoints 104 can be added to improve indoor performance. In areas withpoor wireless coverage, where increased localized performance andaccuracy enhancement is desired, a local wireless access point 104 canbe configured as a DGPS beacon.

An app running on mobile device 101 accesses a GPS module on the device.In some embodiments, the app may also communicate with GPS modules onother devices, such as a GPS module in communication with a local serverrunning, for example, on the set-top box/media gateway 103. In someembodiments, the app on mobile device 101 sends raw location data to aserver that may be remote across the Internet or on set top box 103.This may allow the mobile device to offload certain processing tasks toa more powerful processor to allow greater processing of GPS, AGPS,DGPS, and SDGPS position information. In some embodiments, the mobiledevice may get its first fix geolocation coordinates 801 within about asecond. The mobile device may then the track aGPS coordinates in nearreal-time. Augmented GPS (AGPS) can refine the indoor geocodes 802,while SDGPS processing generates higher precision geocode coordinates(803) in the indoor environment in real time once a lock is established.The mobile device and its mobile app are now ready to interface withnumerous nearby interactive devices, including video displays, smarthome automation and/or security panel 601, smart home lighting switchand dimmer 602, smart home security camera 603, or other devicescontrolled by a home automation server, including lamp 604.

FIG. 12 illustrates various gestures that may be used by users with someembodiments. Three instances of the display that may be provided onvideo screen 102 are illustrated, 610 which shows icons reflecting thegesture detected, 612 which shows a virtual keyboard that allows theuser's gestures to type in information, and a menu 614, which shows theuser selecting various programming from a menu using gestures. In thisexample, gestures can be made via mobile device 101 by either moving thedevice in a swiping motion, such as motion 615 or by swiping a finger onthe touchscreen of device 101. In some embodiments, the screen of mobiledevice 101 may also include a user interface 619 that mirrors orsupplements a menu on video screen 102, such as menu 614.

In a first example, the user points mobile device 101 at TV screen 102to activate a cursor pointing icon shown in UI 610. The user swipes thetouchscreen (617) to move the cursor pointing icon on the TV display,and she may also move the handheld in a swiping gesture (615) to movethe cursor pointing icon on the TV display. She may then tap thetouchscreen to select OK, and also has the option to use an OK gesturemotion by moving the handheld device in space.

In another example, pointing mobile device 101 at the location of TVdisplay area (such as UI 614) can invoke a duplication of the graphicaluser interface (GUI) icons on the display of the handheld wirelessdevice (619). The user then swipes the touchscreen (617) to scrollthrough graphical icons on the TV display and on the handheld wirelessdevice display simultaneously. She may also move the handheld in aswiping gesture (615) to scroll through graphical icons on the TVdisplay and on the handheld wireless device display simultaneously.Navigating media content on a television display is often a socialbehavior among couples or groups, and in this User Experience (UX)configuration, illustrated as replicating the graphical user interface(GUI) icons on the display of the handheld wireless device (619), themembers of the viewing audience group that are not holding mobile device101 have complete visibility and observation of the content icons beingnavigated and displayed on the mobile device 101.

In another example, a user can also interacts with a duplication of thegraphical user interface (GUI) icons on television display 102 and themobile device. In some embodiments, multiple mobile devices can be used,allowing multiple users to simultaneously interact with the screen.Multiple users may be enabled for point-to-interact functionality withthe content navigation User Experience (UX) displayed on the TV screen.

The graphical user interface may be displayed with multiple motifs forpoint-and-click interaction mimicking touchscreen functionality fromacross the room. The user may sit in lean back posture with her handheldwireless device in hand and point at clickable buttons on the TVdisplay, e.g., alphanumeric buttons for entering a search string. TheUser Experience (UX) functionality can be integrated with other readilyavailable inputs i.e., voice and speech recognition via the microphonein the handheld wireless device so that when she says the word “search”,the Search Screen in the UI (612) is invoked on the TV display. In someembodiments, she may also say a search term, such as “pointer,” allowingthe search string field to be quickly populated. If the speechrecognition software mistakenly interprets the word and populates thesearch string field with the word “Painter”, the user can then quicklyand easily point the handheld wireless device at the TV display areawhere she can point-and-click the incorrect spelling in the searchstring to modify the search string with a few quick pointing andtouching motions and gestures to correct the search input and thenselect OK to execute the keyword search.

FIG. 13 illustrates how a user can use some embodiments to interact withvideo playback on a video display. In one example, when a user iswatching a video stream on the mobile device 101, she may point themobile device 101 at the TV screen 102 to transfer or replicate thevideo feed on the TV screen 102 and an icon 622 for user input can bedisplayed on an overlay on the video feed 620. She may select that iconto continue viewing the video feed on the TV screen from the same spotin the frame sequence as was playing on the handheld wireless device.

In another example, a user employs a flip motion gesture to requestdisplay of a linear video channel or on-demand or recorded video contentfeed displayed on the handheld wireless device to be displayed on videodisplay 102. The user then points to click the ‘Resume’ button 622 tocontinue watching the same video program from the same frame sequence inthe source video on TV display 102 that she has playing on the handheldwireless device 101.

Pointing the handheld wireless device 101 at the location of TV displayarea including that of a video projection screen location can beinterpreted as a motion and gesture command that invokes a duplicationof the graphical user interface (GUI) icons on the display of thehandheld wireless device. Onscreen icons based on click-and-holdinteraction (e.g., the video scrubber bar 626) can be replicated on thehandheld wireless device (e.g., as video scrubber bar 627) so thatfunctionality is extended to the television screen, and specificgestures (e.g., rotate up) can invoke key video watching commands (e.g.,pause, play, and stop video). The user employs a touchscreenclick-hold-and-drag motion on the handheld wireless device to controlthe DVR scrubber bar icon on the TV display for slider control overpause/fast-forward/rewind DVR functions.

FIG. 14 illustrates how a user can use some embodiments to interact withnon-video interactive devices, such as IP-enabled security systems thatinteract with a home automation server or have home automation serversoftware embedded that allow the security system to be remotelycontrolled and/or remotely control other devices, such as electricaloutlets or lights. Pointing the handheld wireless device 101 at smartcontrol panel 630, allows the mobile device to be configured as a motionand gesture command device that invokes a duplication of the userinterface presented on the smart control panel (UI 632, replicated as UI634). The replication of the UI 632 and 634 need not be an exactreplica. For example, UI 632 may include a touchpad having more or lessfunctionality than an icon-driven touch screen UI 634 on device 101.

Changing the orientation (such as by movement 636) of the handheldwireless device to be directionally and tilt angle aligned with theknown location of smart light switch 640 by the user can invoke acommand to activate a duplication of the user interface 642 presented onthe smart light switch. The user now uses the touchscreen interface644of the handheld wireless device to interact with the buttons on thelight switch. Again, the replication can be functional, rather thanstylistic, allowing some or all buttons to be replicated on the mobiledevice to allow the user to operate the switch from her mobile device.

FIG. 15 illustrates the algorithmic logic flow of calculationinstructions used to implement RTSM localization in some embodiments ofthe present invention. The steps shown in FIG. 15 may be carried out byan app on mobile device 101, server software on a remote server or aset-top box, or any combination thereof. The diagram represents acomputational process that may be ongoing and repeating during theoperation of the mobile app 201. The calculations are described as aReal-Time Serial Multilateration (RTSM) algorithm. At step 650, clientsoftware 201 (described as a pointer app) initiates the RTSM algorithmto refine precision high-resolution geocodes for the localization of themobile device 101. At step 653, the software gets geocode inputs for thelocation of the mobile device 101 in the form of either DGPS correctedgeocode coordinates, or AGPS geocode coordinates that have been refinedwith coarse localization techniques using Time of Arrival measurementsfor radio signal transmissions to one or more cellular towers withinrange. The resolution of the input coordinates are expected to be 1meter vertical and roughly 0.3-1.0 meters horizontal depending on thelatitude at the location of the measurements, with an accuracy ofplus-or-minus 5 meters. At step 656, the geocode values are convertedinto a high-resolution format for four key coordinate variables plus onesupplemental variable; latitude (ddmm.mmmmmm), longitude(dddmm.mmmmmmm), Altitude above mean sea level (m.mmm), Height abovegeoid centroid (m.mmm), and Elevation above ground level (m.mmm) Thecoordinate conversation in step 656 normalizes for inconsistencies inthe formatting of input values sourced from multiple aGPS and AGPS,including DGPS and SDGPS, systems and processes, and adds decimal placesfor granularity down to millimeters of localization resolution, alongwith adding placements for one to two key variables that will assistwith determining a precise elevation of the mobile device 101 duringoperation of the mobile app 201.

At step 659, the calculations use a predefined process that accessesstored data and additional inputs to begin high-resolution refinement ofthe geocodes for real time localization on the mobile device 101. Theprocess cross references a stored terrain elevation model to initiallypopulate the Elevation field based on the Latitude and Longitude inputs,and if the Height value is null accesses a stored geoid model of theearth to populate the Height field based on the Latitude and Longitudeand Altitude inputs. Altitude and Height are the least accurate of theGPS-based coordinates used for inputs, and precision high-resolutionElevation measurements are valuable to the operation of the mobiledevice 101 pointing mechanism in three dimensional space, so in someembodiments a serial learning algorithm is employed to reduce variancein the accuracy of the Elevation values used in the RTSM algorithm.

Once the five variables in step 659 (X, Y, Z, H, E) have been populated,in some embodiments the predefined process invokes a triangulationcalculation to refine the values for better accuracy based on distancemeasurements to cellular towers within range using TDOA (time differenceof arrival) of the radio frequencies send from the mobile device 101 tothe cellular tower site 107. In some embodiments the predefined process'triangulation calculation also accounts for angle of approachmeasurements to cellular towers within range using AOA (angle ofarrival) of the radio frequencies sent from the mobile device 101 to thecellular tower site 107. At completion of step 659 the accuracy of thegeocode measurement can be better than ˜2 meters horizontal and ˜3meters vertical.

At step 662, the RTSM algorithm applies a bounded range filter, the1^(st) Standard Deviation Filter in the RTSM algorithm, on the set ofthe (X, Y, Z, H, E) variable combinations identified by the calculationsin step 659 as viable high-resolution location possibilities for thetime period of the localization measurement which may be within aspecific 50 millisecond time sequence. The standard deviation filtercreates a bounded range across the sample set of geocode combinations toeliminate outlier values that may have been introduced by signal noiseor interference with the radio signals used for distance and anglemeasurements.

At step 665, a first multilateration calculation is applied to themobile device's 101 radio communications with cellular towers withinrange using TDOA. Multilateration is a common technique in radionavigation known as hyperbolic navigation. Multilateration does notrequire a common clock, and the difference in the signal timing can bemeasured with an oscilloscope to determine absolute distance betweenradio transmitter and receiver, however multilateration does require oneradio transmitter to be in radio communication with four or more radioreceivers during the time frame of the localization measurements todetermine a location in three dimensional space. Multilateration can bemore accurate than triangulation, but can also be more computationallyintensive, because it calculates and the intersection of at least threehyperboloid geometries in three dimensional space. In some embodiments,the mobile app 201 sends and receives only the values used formultilateration computations to a server to offload processing of thecalculations used in the RTSM algorithm. At completion of step 665, theaccuracy of the geocode measurement can be better than ˜1 meterhorizontal and ˜1.5 meters vertical.

At step 668, a second multilateration calculation is performed based onthe mobile device's 101 radio communications with wireless data accesspoints 453 within range using TDOA. Multilateration using Wi-Fi benefitsfrom four or more wireless data receivers at known locations withinrange of the mobile device 101. In some embodiments wireless accesspoints can be added with a registration process that publishes theirknown location in high-resolution geocode format for reference in thelocalization process using the RTSM algorithm. At completion of step668, the accuracy of the geocode measurement can be better than ˜0.5meters horizontal and ˜1 meter vertical.

At step 671, a third multilateration calculation is performed based on acombination of the mobile device's 101 radio communications withcellular towers within range and with wireless data access points 453within range using TDOA. In some embodiments both types of radioreceivers (GSM or CDMA, and Wi-Fi) are included to complete thenecessary four or more receivers needed for a single multilaterationcalculation. In other embodiments, TDOA distance measurement accuracyattributes are applied to each receiver site used in the multilaterationcalculation based on the radio propagation characterizes of the localenvironment, and the wireless signal type and frequency used by eachreceiver. At completion of step 671 the accuracy of the geocodemeasurement can be better than ˜0.1 meters horizontal and ˜0.3 metersvertical.

At step 674, a stored procedure applies bounded weightings to thesamples collected and logged for the time period of the localizationmeasurement. In some embodiments a Cramér-Rao bound or similar boundedrange is used to apply probability weightings to the viable accuracy ofthe high-resolution geocode combinations collected in steps 665, 668,and 671.

At step 677, a second stored procedure is used to analyze the samplesand weightings logged through step 674, and then apply historicallearning to the accuracy parameters to expresses upper and lower boundson the variance distribution of the localization geocode combinations toproduce a resulting probabilistic geocode for the time frame of thelocalization measurements.

At step 680, a rule is applied to the result from step 677 based on theprobability confidence in the geocode coordinates produced for thelocalization of the mobile device 101 in the sub-one-second real timeduration of the measurement for variances within plus-or-minus 1.5 cmvertical and horizontal. In some embodiments the rule can be modifiedfor higher or lower localization resolution or accuracy, or coarser ornarrower time durations of measurements, depending on the use cases ofthe user interactivity and usability factors of the mobile device's 101pointing and gesturing user interface. If the rule in step 680 ispassed, the geocode result is deemed sufficient for the use case and ispassed on for output.

If the rule in step 680 is not passed, additional serial calculationsthrough the learning algorithm may be needed to generate the usableresult(s). In step 686, clock synchronization is applied to the TDOA,which may be at the sub-nanosecond level of granularity. For radiocommunications within the same network, clock synchronization is morepracticable. While multilateration does not require clocksynchronization, nanoseconds of clock drift from one set of receivertime differences to the next can cause multiple centimeters of variancein the result. Electromagnetic environmental conditions can also changeat the millisecond and nanosecond levels adding variance to the results.Clock synchronization is an additional supplemental step in the RTSMalgorithm that adds accuracy at high resolutions, and can generateadditive samples for use in the learning algorithm that can improveestimation of a deterministic parameter. In some embodiments, clocksynchronization is used to improve the user experience.

At step 689, a determination is made regarding which radio receivers areeligible for clock synchronization. If cross-network clocksynchronization is available, meaning that Wi-Fi and Cellular radioreceivers are connected to the same or interconnected carrier networksin a way that allows for or enables clock synchronization across boththe cellular and wireless data networks, then the process forcalculations returns to step 665, Multilateration1 starting again withsignaling across the cellular network and progressing through steps 668,and 671 to sample TDOA signaling on also the wireless data network, andfinally a hybrid combination of both cellular and wireless datareceivers. If clock synchronization is not available, or is onlyavailable in one portion of the local wireless network, then the processfor calculations returns to step 668, Multilateration2 sampling thelocal wireless data, and step 668 inserts those results into step 674for enhancements to the bounded weightings. If after a second setMultilateration geocodes have been generated the results provided bystep 677 still fail step 680, additional attempts can be made passingthrough external procedure 692.

Step 692 represents procedures that are external to the RTSM algorithmthat address limitations in the local environment wirelessinfrastructure that might be temporary, intermittent, or requiremodification. Solutions for these limitations may or may not beachievable within the real-time duration of the measurements, describedin some embodiments at <50 milliseconds. Examples of some of theseexternal procedures are upgrading some of the radio receiver siteswithin range to use high-resolution geocode formats, identifyingadditional radio receiver sites within range that do not have knownpublished locations and looking up approximate location references forthem, or requesting the known locations of those sites be published, oradding known location information to those wireless sites in a databaseor in their manifest. Continuous processing of the RTSM algorithm whilethese external procedures are followed can increase the algorithm'slearning of the natural features and radio frequency physics of thelocal environment making that outputs at step 683 are generated. In someembodiments non-real-time outputs at step 683 that are performanceconstrained while the external procedures 692 are followed may be usedto set-up or calibrate the overall system prior to operation by the enduser. In other embodiments that do not require real-time localization,the overall system may be operated using the outputs from step 683 asthey are generated. In still other embodiments the RTSM algorithm is anoptional augment to the already available DGPS and AGPS coordinates tothe wireless device 101.

In step 695, the timestamps of the last two results generated as outputin step 683 are compared to see if they occurred more than 40milliseconds apart. If they did, the RTSM algorithm process is repeated,whenever in real-time operation using the mobile app 201 for pointingand gesturing interaction. If they occurred within 40 milliseconds ofeach other, the calculations wait at step 698 until the next iteration.

Although the invention has been described with reference to exemplaryembodiments, it is not limited thereto. Those skilled in the art willappreciate that numerous changes and modifications may be made to thepreferred embodiments of the invention and that such changes andmodifications may be made without departing from the true spirit of theinvention. For example, references to Wi-Fi are intended to refer tocurrent 802.11 standards, as well as future variations, including otherwireless IP protocols. It is therefore intended that the appended claimsbe construed to cover all such equivalent variations as fall within thetrue spirit and scope of the invention.

1-30. (canceled)
 31. A system for remotely interacting with aninteractive device having a display, comprising: computer-readableinstructions for configuring a processor of a handheld wireless device,the handheld wireless device being separate from the interactive device,and the processor of the handheld wireless device determining locationand orientation information about the handheld device based on sensorsonboard the handheld wireless device and sending the locationinformation across at least one wireless network; at least one remoteserver configured to receive the location information and to determinethe location of the handheld wireless device relative to the interactivedevice, based on the location information received from the handhelddevice and a model of the environment around the handheld wirelessdevice, wherein the location information is derived by the handheldwireless device from a distance measurement between the handheldwireless device and a plurality of beacons, including at least onefixed-location terrestrial wireless networking device, and wherein theserver is configured to remotely change a user interface of theinteractive device in response to the position information; andcomputer-readable instructions for duplicating, on a display of thehandheld device, at least a portion of information displayed on theinteractive device, wherein the user interface of the handheld wirelessdevice allows user interaction with the interactive device and theinformation displayed on the user interface varies based on at least oneof the location and orientation of the handheld wireless device.
 32. Thesystem of claim 31, further comprising at least one of a set-top boxapp, or Smart TV app, in communication with the at least one server,that displays the user interface of home or building automationcomponents on at least one of a video display and the handheld device,and updates the user interface in response to changes in the locationand orientation information.
 33. The system of claim 31, wherein the atleast one home or building automation components are elements of eitherof a home security system or a public safety system, comprising at leastone of a security camera, and a security panel.
 34. The system of claim31, further comprising computer-readable instructions that determinemotion information relating to the location and orientation informationto recognize gestures made by a user.
 35. The system of claim 31,further comprising computer-readable instructions to allow a processorto recognize at least a subset of the following gestures: pointing atthe interactive device, swiping at an icon on a digital display,flipping through icons or content on a digital display, flipping aswitch, scrolling across or down a slider icon, and pushing a digital orphysical button.
 36. The system of claim 31, wherein locationinformation is further derived from GPS signals received by the handhelddevice and the location information is further refined by distancemeasurements to at least one fixed-location terrestrial wirelessnetworking device comprising at least one of a Wi-Fi access point, and acellular base station and the location information corresponds to anindoor location,
 37. The system of claim 31, wherein the user interfaceallows for identifying and registering new separate interactive devicelocations, and presents an overlay of duplicate menu or icon elementsonto the mobile device as the device is moved past the interactivedevice location in physical space, using a model of the physical space,to facilitate interaction with a plurality of separate interactivedevices, and to determine which of the plurality of interactive devicethe user interacts with based on the location and orientationinformation.
 38. The system of claim 31, wherein the handheld wirelessdevice comprises one of a tablet computer and a cell-phone.
 39. A systemfor facilitating remote interaction with interactive devices in aphysical space, comprising: a first set of computer-readableinstructions that configure a handheld wireless device to determinelocation and orientation information about the handheld wireless devicebased on sensors onboard the handheld wireless device and transmit thelocation information and at least one user input across a wirelessnetwork, wherein the location information is derived by the handheldwireless device from a distance measurement between the handheldwireless device and at least one fixed-location terrestrial wirelessnetworking device; a server, remote from the handheld wireless deviceand in communication with the handheld wireless device, configured toexecute: a second set of computer-readable instructions that configurethe server to receive the location information and to approximate theorientation of the handheld wireless device relative to a physicalspace, including in relation to at least one interactive device, whichis separate from the handheld wireless device, based on the locationinformation received from the handheld device and a model of theenvironment around the handheld wireless device and a third set ofcomputer-readable instructions that configure the server to facilitatenetwork control of the at least one interactive device in the physicalenvironment in response to the location information and the at least oneuser input as the handheld device is moved past the separate interactivedevice locations in physical space, using a model of the physical space;and a fourth set of computer-readable instructions that configure ahandheld wireless device to duplicate on a display of the handhelddevice at least one a portion of information displayed on the separatecontrolled interactive device.
 40. The system of claim 39, wherein theseparate interactive device comprises at least one of an interactivevideo display.
 41. The system of claim 39, wherein the server is anautomation server and the at least one interactive device comprises atleast one of a light switch, an alarm device, and an HVAC devicecontrolled by the automation server.
 42. The system of claim 39, whereinthe first set of computer-readable instructions comprises instructionsfor displaying, on a display of the handheld wireless device, selectableactions that can be taken to interact with the at least one interactivedevice to solicit the at least one user input.
 43. The system of claim39, wherein the selectable actions displayed on the handheld wirelessdevice change depending on which of a plurality of types of interactivedevices that the handheld wireless device points to.
 44. The system ofclaim 39, wherein location information is further derived from at leastone of GPS signals, received by the handheld device and the locationinformation is further refined by distance measurements to at least onefixed-location terrestrial wireless networking device comprising atleast one of a Wi-Fi access point and a cellular base station receivedby the handheld device and the location information includes horizontaland vertical location information.
 45. A method for facilitatinginteraction between a mobile device and interactive devices comprisingsteps of: a. observing by a processor on a handheld wireless device,utilizing an antenna and one or more sensors on the handheld wirelessdevice, a plurality of distances relative to a plurality offixed-location terrestrial network devices in a physical space and anorientation of the handheld wireless device to form the location andorientation information that reflects the location and orientation ofthe handheld wireless device within the physical space, and transmittingthe location information from the handheld wireless device across atleast one network; b. receiving at a remote server, across the at leastone network, the location information from the handheld wireless device;c. calculating, by a processor at the remote server, a location of thehandheld wireless device relative to at least one interactive devicefrom the location and orientation information and a model of thephysical space to determine that a user is pointing the handheldwireless device relative to the interactive device; d. determining, byat least one of the processors, at least one command input from a userof the handheld wireless device; e. transmitting the command to the atleast one interactive device across at least one of the at least onenetworks in response to the step of determining and the locationinformation; and f. duplicating, on a display of the handheld device, atleast a portion of the user interface information of the interactivedevice, wherein the at least one interactive device comprises acontrollable element in the physical environment that can be controlledas the handheld device is moved past the separate interactive devicelocations in physical space, using a model of the physical space. 46.The method of claim 45, wherein the separate interactive devicecomprises a controllable element in the physical environment that can becontrolled as the device is moved past the separate interactive devicelocations in physical space, using a model of the physical space. 47.The method of claim 45, wherein the at least one interactive devicecomprises at least one Internet-connected component consisting of atleast one of a light switch, a light bulb, an alarm device, a securitypanel, security camera, a home or building automation panel, an HVACdevice, or other devices controlled by an automation server.
 48. Themethod of claim 45, wherein location information is further derived fromat least one of GPS signals received by the handheld device and thelocation information is further refined by distance measurements to atleast one fixed-location terrestrial wireless networking device.
 49. Themethod of claim 45, further comprising a step of calibrating thelocation of the at least one interactive device by providingcomputer-readable instructions to allow the handheld wireless device torecord a location of at least one portion of the at least oneinteractive device when the handheld wireless device is placedapproximately at the location of the interactive device in the physicalspace.
 50. The method of claim 45, wherein the step of determining atleast one command input comprises detecting a motion gesture made by auser of the handheld wireless device.